Detection based on fusion of multiple sensors

ABSTRACT

A system and method to perform detection based on sensor fusion includes obtaining data from two or more sensors of different types. The method also includes extracting features from the data from the two or more sensors and processing the features to obtain a vector associated with each of the two or more sensors. The method further includes concatenating the two or more vectors obtained from the two or more sensors to obtain a fused vector, and performing the detection based on the fused vector.

INTRODUCTION

The subject disclosure relates to detection based on the fusion ofmultiple sensors.

Vehicles (e.g., automobiles, trucks, farm equipment, constructionequipment, automated factory equipment) are increasingly instrumentedwith sensors facilitate augmented or automated operation. Exemplarysensors include those that capture data about the environment around thevehicle and those that capture data about the vehicle. For example,cameras, audio detectors (e.g., microphones), and radar or lidar systemsobtain data about the environment around the vehicle (e.g., otherobjects in the vicinity of the vehicle). As another example,accelerometers, speedometers, and the like obtain data about the vehicleand its operation. In prior systems, fusion of information fromdifferent sensors for purposes of detection involved fusion of thedetection results. That is, each individual sensor determined aconfidence level associated with detection, and the results from two ormore sensors were combined to make a detection determination. However,this detection architecture cannot fully take advantage of informationfrom each sensor. Accordingly, it is desirable to provide detectionbased on fusion of multiple sensors.

SUMMARY

In one exemplary embodiment, a method to perform detection based onsensor fusion includes obtaining data from two or more sensors ofdifferent types. The method also includes extracting features from thedata from the two or more sensors and processing the features to obtaina vector associated with each of the two or more sensors. The methodfurther includes concatenating the two or more vectors obtained from thetwo or more sensors to obtain a fused vector, and performing thedetection based on the fused vector.

In addition to one or more of the features described herein, the methodalso includes normalizing each of the two or more vectors associatedwith the two or more sensors prior to the concatenating.

In addition to one or more of the features described herein, the methodalso includes normalizing the fused vector prior to the performing thedetection.

In addition to one or more of the features described herein, theperforming the detection includes implementing a machine learningalgorithm.

In addition to one or more of the features described herein, theperforming the detection includes implementing a rule-based algorithm.

In addition to one or more of the features described herein, theobtaining the data from the two or more sensors includes obtaining thedata from a microphone and a camera.

In addition to one or more of the features described herein, theobtaining the data from the two or more sensors includes obtaining thedata in a vehicle.

In addition to one or more of the features described herein, theperforming the detection includes detecting a rumble strip using thefused vector based on the two or more sensors being in the vehicle.

In another exemplary embodiment, a system to perform detection based onsensor fusion includes two or more sensors of different types to obtaindata. The system also includes a processor to extract features from thedata from the two or more sensors, process the features to obtain avector associated with each of the two or more sensors, concatenate thetwo or more vectors obtained from the two or more sensors to obtain afused vector, and perform the detection based on the fused vector.

In addition to one or more of the features described herein, theprocessor normalizes each of the two or more vectors associated with thetwo or more sensors prior to concatenating.

In addition to one or more of the features described herein, theprocessor normalizes the fused vector prior to performing the detection.

In addition to one or more of the features described herein, theprocessor is configured to perform the detection by implementing amachine learning algorithm.

In addition to one or more of the features described herein, theprocessor implements a rule-based algorithm.

In addition to one or more of the features described herein, the two ormore sensors includes a microphone and a camera.

In addition to one or more of the features described herein, the two ormore sensors are in a vehicle.

In addition to one or more of the features described herein, theprocessor detects a rumble strip.

In another exemplary embodiment, a lane departure system in a vehicleincludes a camera to obtain images, and a microphone to obtain audiodata. The system also includes a controller to extract visual featuresfrom the images, extract audio features from the audio data, combine thevisual features and the audio features into combined features, performdetection based on the combined features, and indicate lane departurebased on the detection.

In addition to one or more of the features described herein, the systemalso includes an inertial measurement unit (IMU) configured to obtainvibration data. The combined features include features extracted fromthe vibration data.

In addition to one or more of the features described herein, thedetection detects a rumble strip indicating a shoulder area of aroadway.

In addition to one or more of the features described herein, thecontroller performs augmented or automated vehicle action based on thedetection.

The above features and advantages, and other features and advantages ofthe disclosure are readily apparent from the following detaileddescription when taken in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, advantages and details appear, by way of example only,in the following detailed description, the detailed descriptionreferring to the drawings in which:

FIG. 1 is a block diagram of a system to perform detection based on thefusion of multiple sensors according to one or more embodiments;

FIG. 2 illustrates several rumble strips that may be detected accordingto one or more embodiments;

FIG. 3 shows a process flow of a method to perform detection based onthe fusion of multiple sensors according to one or more exemplaryembodiments; and

FIG. 4 is a process flow of an exemplary method of performing rumblestrip detection based on the fusion of multiple sensors according to oneor more embodiments.

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is notintended to limit the present disclosure, its application or uses. Itshould be understood that throughout the drawings, correspondingreference numerals indicate like or corresponding parts and features.

As previously noted, vehicles include multiple sensors and sensor fusiongenerally combines information from multiple sensors. Currently,detection based on sensor fusion refers to combining detection resultsfrom multiple sensors. That is, detection results and correspondingprobabilities or confidence levels are provided by various sensors. Theoverall detection result may then be obtained through a rule-basedcombination or averaging of the detection results.

Embodiments of the systems and methods fuse the information from thevarious sensor prior to performing detection. Specifically, data fromeach sensor is concatenated prior to implementing a detection algorithm.The detection algorithm may be implemented as a machine learningalgorithm, for example. The learning considers data from each sensorsuch that the system performs detection based on information from everysensor rather than being provided with detection results from everysensor as in prior systems.

A specific detection scenario is discussed herein. Three types ofsensors, a camera, a microphone, and an inertial measurement unit (IMU)are used to detect a rumble strip and identify its location. Thisdetection may be part of a fully or partially autonomous vehicleoperation. Rumble strips may be located on the shoulder to indicate thata vehicle is leaving the roadway. Rumble strips may also be located longa line that separates lanes travelling in different directions.Transverse rumble strips may be located in areas (e.g., highway offramps, prior to stop signs or traffic lights) to indicate that a vehicleshould slow down or stop. While the rumble strip detection scenario isdiscussed for explanatory purposes, the fusion-prior-to-detectionarchitecture detailed herein is applicable to other fusion detection oridentification scenarios and is not intended to limit the applicabilityof the architecture.

In accordance with an exemplary embodiment, FIG. 1 is a block diagram ofa system to perform detection based on the fusion of multiple sensors105. The exemplary vehicle 100 in FIG. 1 is an automobile 101. Threeexemplary sensors 105 are indicated. One type of sensor 105 is a camera110. Four cameras 110 are shown at different locations around thevehicle 100. Another type of sensor 105 is a microphone 120. Fourmicrophones 120 are shown in FIG. 1. The third type of sensor 105 shownin FIG. 1 is an IMU 130. The IMU 130 may include gyroscopes andaccelerometers and measure vibration of the vehicle 100. Each of thesetypes of sensors 105 is known and not further detailed herein.

A controller 140 performs the detection based on information from thedifferent sensors 105. The controller 140 may implement a machinelearning algorithm, as further discussed. The controller 140 includesprocessing circuitry that may include an application specific integratedcircuit (ASIC), an electronic circuit, a processor (shared, dedicated,or group) and memory that executes one or more software or firmwareprograms, a combinational logic circuit, and/or other suitablecomponents that provide the described functionality. The controller 140may be in communication with an electronic control unit (ECU) 150 of thevehicle 100 that controls various vehicle systems 170 (e.g., collisionavoidance, adaptive cruise control, autonomous driving). In alternateembodiments, the controller 140 may be part of the ECU 150. In additionto the ECU 150 and sensor 105, the controller 140 may be incommunication with other systems 160 such as a global positioning sensor(GPS) system or mapping system. The other systems 160 may also includethe infotainment system that obtains inputs from a driver and providesoutput to the driver.

FIG. 2 illustrates several rumble strips 210 that may be detectedaccording to one or more embodiments. Rumble strips 210 may be raisedbars or grooves in the roadway. FIG. 2 shows a vehicle 100 a travellingin a lane 220 of a roadway and another vehicle 100 b travelling inanother lane 225 in the opposite direction as vehicle 100 a. Two sets ofrumble strips 210 a may be used to delineate the lanes 220 and 225 withtraffic travelling in opposite directions. A set of rumble strips 210 bdelineates the lane 220 of the roadway from the shoulder 230. Transverserumble strips 210 c are shown ahead of the vehicle 100 a. These rumblestrips 210 c are generally arranged in periodic sets and may indicate anupcoming stop or area of slower speed, for example. According to theperiodic arrangement shown in FIG. 2, the vehicle 100 a encounters threesets of four rumble strips 210 c. One or both of the vehicles 100 a, 100b may include the sensors 105 and controller 140 discussed withreference to FIG. 1 to perform detection based on the fusion of multiplesensors 105.

FIG. 3 shows a process flow of a method to perform detection based onthe fusion of multiple sensors 105 according to one or more exemplaryembodiments. The exemplary embodiments relate to detecting rumble strips210 based on the fusion of sensors 105 that include cameras 110,microphones 120, and an IMU 130. As previously noted, while theexemplary embodiments are detailed for explanatory purposes, the fusionarchitecture discussed with reference to FIG. 3 is applicable to thefusion of additional or alternate sensors 105 and to the detection ofalternate or additional features to rumble strips 210.

At blocks 310 a, 310 b, 310 c, the processes respectively includeobtaining camera data from the cameras 110, obtaining microphone datafrom the microphones 120, and obtaining IMU data from the IMU 130. Atblocks 320 a, 320 b, 320 c, the processes include feature extraction.

At block 320 a, extracting visual features from the data obtained fromthe cameras 110 is a known process and may include a series of processesto refine extracted features. The series of processes may includeperforming low-level feature extraction, performing mid-level featureextraction on the low-level features, and then performing high-levelfeature extraction on the mid-level features. For example, a multilayerconvolutional neural network may be used to extract features from theimages captured by the camera 110. At block 320 b, extracting audiofeatures from the data obtained from the microphones 120 is also knownand refers to obtaining a vector of microphone levels, for example. Atblock 320 c, extracting acceleration features from the data obtainedfrom the IMU 130 also refers to obtaining a vector of values accordingto a known process.

At blocks 330 a, 330 b, 330 c, the processes include normalizing thefeatures that are respectively extracted at blocks 320 a, 320 b, and 320c. Normalization may refer to a number of different operations such asre-scaling, re-sizing vector lengths, and other established techniques.For example, normalizing the extracted features, at each of the blocks330 a, 330 b, 330 c, may result in a vector of values from −1 to 1.Thus, the image features from feature extraction at block 320 a isnormalized to a vector of values from −1 to 1, and the feature vectorsobtained at blocks 320 b and 320 c are also normalized to values from −1to 1. These processes are unique to the architecture detailed herein. Inprior fusion systems, extracted features from data obtained with each ofthe sensors 105 is used to perform detection individually. According toembodiments of the architecture, the vectors obtained at blocks 330 a,330 b, 330 c are combined prior to performing detection, as detailed.

Concatenating, at block 340, refers to concatenating the vectorsobtained by the normalizing at blocks 330 a, 330 b, and 330 c. Theresult of concatenating is a vector with values between −1 and 1 that isthe combined length of each of the vectors obtained at blocks 330 a, 330b, and 330 c. Normalizing, at block 350, refers to a normalization ofthe concatenated vector obtained at block 340. As previously noted,normalization may refer to re-scaling, re-sizing, or otherwisemanipulating the concatenated vectors (e.g., parametric normalization,quantile normalization). The normalization at block 350 may be adifferent type of normalization than the normalization at blocks 330 a,330 b, 330 c. Performing detection, at block 360, refers to determiningwhether the concatenated data indicates the presence of rumble strips210 and determining a confidence level of the detection. The detectionmay include implementing a machine learning algorithm and training thealgorithm based on the data from all three types of sensors 105. Aspreviously noted, a different set of sensors 105 may be fused to detecta different feature according to alternate embodiments.

FIG. 4 is a process flow of an exemplary method of performing rumblestrip 210 detection based on the fusion of multiple sensors 105according to one or more embodiments. At block 410, detecting the rumblestrip 210 refers to the controller 140 obtaining the result of theprocessing at block 360, as discussed previously. A determination isthen made, at block 420, of whether the rumble strip 210 relates to alane divider (i.e., rumble strips 210 a or 210 b), as opposed totransverse rumble strips 210 c.

This determination may be made according to different embodiments. Forexample, the periodicity of rumble strips 210 c may be used todistinguish them from lane departure-alerting rumble strips 210 a, 210b. This is because, as noted previously and illustrated in FIG. 2, thetransverse rumble strips 210 c are not continuous like rumble strips 210a, 210 b but, instead, are periodic. As another example, the length ofthe rumble strips 210 c may be used to make the determination at block420. The transverse rumble strips 210 c cause the rumble sound to bepicked up by microphones 120 on both sides of the vehicle 100 and arevisible on cameras 110 on both sides of the vehicle 100.

If the rumble strip 210 is determined, at block 420, not to relate to alane divider, then it is assumed to be a transverse rumble strip 210 c.In this case, at block 430, performing correction or providing a messagerefers to the controller 140, directly or through the ECU 150,instructing vehicle systems 170 to slow or stop the vehicle 100 orproviding an alert to the driver through one of the other systems 160(e.g., infotainment screen), for example. If the rumble strip 210 isdetermined, at block 420, to relate to a lane divider, then a set ofprocesses is implemented to take corrective action.

At block 440, detecting each rumble strip 210 a, 210 b refers to usingthe relevant cameras 110. At block 450, estimating the heading anddistance to the rumble strip 210 edge refers to determining the headingand distance to the closer edge of the rumble strips 210. Based on theestimate at block 450, estimating the steering correction angle, atblock 460, refers to the angle to put the vehicle 100 back in thecorrect lane (e.g., 220). Estimating the steering correction angle, atblock 460, includes determining the road curvature, at block 465. Thedetermination of road curvature, at block 465, may include using one ofthe other systems 160 like the mapping system, for example. The steeringcorrection angle estimated at block 460 may be used, at block 470, toperform automated action or to provide information to the driver. Atblock 470, performing correction refers to the controller 140controlling vehicle systems 170 directly or through the ECU 150. Atblock 470, providing a message refers to the controller 140 alerting thedriver to the required correction.

While the above disclosure has been described with reference toexemplary embodiments, it will be understood by those skilled in the artthat various changes may be made and equivalents may be substituted forelements thereof without departing from its scope. In addition, manymodifications may be made to adapt a particular situation or material tothe teachings of the disclosure without departing from the essentialscope thereof. Therefore, it is intended that the present disclosure notbe limited to the particular embodiments disclosed, but will include allembodiments falling within the scope thereof

What is claimed is:
 1. A method to perform detection based on sensorfusion, the method comprising: obtaining data from two or more sensorsof different types; extracting features from the data from the two ormore sensors and processing the features to obtain a vector associatedwith each of the two or more sensors; concatenating, using a processor,the two or more vectors obtained from the two or more sensors to obtaina fused vector; and performing the detection, using the processor, basedon the fused vector.
 2. The method according to claim 1, furthercomprising normalizing each of the two or more vectors associated withthe two or more sensors prior to the concatenating.
 3. The methodaccording to claim 1, further comprising normalizing the fused vectorprior to the performing the detection.
 4. The method according to claim1, wherein the performing the detection includes implementing a machinelearning algorithm.
 5. The method according to claim 1, wherein theperforming the detection includes implementing a rule-based algorithm.6. The method according to claim 1, wherein the obtaining the data fromthe two or more sensors includes obtaining the data from a microphoneand a camera.
 7. The method according to claim 1, wherein the obtainingthe data from the two or more sensors includes obtaining the data in avehicle.
 8. The method according to claim 7, wherein the performing thedetection includes detecting a rumble strip using the fused vector basedon the two or more sensors being in the vehicle.
 9. A system to performdetection based on sensor fusion, the system comprising: two or moresensors of different types configured to obtain data; and a processorconfigured to extract features from the data from the two or moresensors, process the features to obtain a vector associated with each ofthe two or more sensors, concatenate the two or more vectors obtainedfrom the two or more sensors to obtain a fused vector, and perform thedetection based on the fused vector.
 10. The system according to claim9, wherein the processor is further configured to normalize each of thetwo or more vectors associated with the two or more sensors prior toconcatenating.
 11. The system according to claim 9, wherein theprocessor is further configured to normalize the fused vector prior toperforming the detection.
 12. The system according to claim 9, whereinthe processor is configured to perform the detection by implementing amachine learning algorithm.
 13. The system according to claim 9, whereinthe processor is configured to perform the detection by implementing arule-based algorithm.
 14. The system according to claim 9, wherein thetwo or more sensors includes a microphone and a camera.
 15. The systemaccording to claim 9, wherein the two or more sensors are in a vehicle.16. The system according to claim 9, wherein the processor is configuredto perform the detection by detecting a rumble strip.
 17. A lanedeparture system in a vehicle, comprising: a camera configured to obtainimages; a microphone configured to obtain audio data; and a controllerconfigured to extract visual features from the images, extract audiofeatures from the audio data, combine the visual features and the audiofeatures into combined features, perform detection based on the combinedfeatures, and indicate lane departure based on the detection.
 18. Thesystem according to claim 17, further comprising an inertial measurementunit (IMU) configured to obtain vibration data, wherein the combinedfeatures include features extracted from the vibration data.
 19. Thesystem according to claim 17, wherein the detection detects a rumblestrip indicating a shoulder area of a roadway.
 20. The system accordingto claim 17, wherein the controller performs augmented or automatedvehicle action based on the detection.